Digital signal processing array using integrated processing elements

ABSTRACT

Techniques and mechanisms described herein include a signal processor implemented as an overlay on a field-programmable gate array (FPGA) device that utilizes special purpose, hardened intellectual property (IP) modules such as memory blocks and digital signal processing (DSP) cores. A Processing Element (PE) is built from one or more DSP cores connected to additional logic. Interconnected as an array, the PEs may operate in a computational model such as Single Instruction-Multiple Thread (SIMT). A software hierarchy is described that transforms the SIMT array into an effective signal processor.

TECHNICAL FIELD

The present disclosure relates generally to the configuration ofcomputer processors, and more specifically to the configuration of fieldprogrammable gate arrays and application specific integrated circuitsfor the parallel processing of digital signals.

DESCRIPTION OF RELATED ART

In recent years, field programmable gate arrays (FPGAs) have emerged asa power-efficient alternative to graphics processing units (GPUs) andmultiple-core central processing units (CPUs) for the implementation ofcomplex algorithms in real-time or near real-time applications. In its2015 investor meeting, Intel estimated that FPGAs will be used in 30% ofdata center servers by 2020. The main drawback to using FPGAs has beenthe difficulty involved in designing complex logic with basic hardwaredescription languages (HDL) and the slow and laborious cycle betweendesign and testing. The introduction of high-level synthesis (HLS) hassomewhat mitigated these challenges, but the issues of long synthesis,long place and route times, and cryptic, machine-generated code stillremain.

One approach to avoiding cryptic code and time-consuming developmentcycles is to use an overlay. Overlay architectures isolate the user evenfurther from the complexity of FPGA synthesis and effectively transforman FPGA implementation into a software production enterprise. Effectivecomputing platforms have been obtained by replicating simple processingelements (PEs) in the FPGA fabric and providing an instruction stream toenable the PEs to operate in a single instruction multiple data (SIMD)configuration.

One example of such an approach is the soft-core processor Octavo, whichhas been used as a building block for a replicated, point-to-point meshof PEs (fine grained). Further enhancements to this architecture usehardened DSP's in the FPGA as co-processors to increase computationalperformance (coarse grained).

More recent representative examples of coarse-grained FPGA overlaysinclude iDEA, FSE, DeCO, and IPPro. These processors include PEs basedon Xilinx DSP cores and are designed to connect with each other and formarrays capable of limited SIMD parallelism. However, due to their costlyconnection and data alignment logic requirements, a typical arrayimplemented in FPGA is unable to use a large percentage of the FPGA DSPresources available and typically does not exceed a modest number ofPEs, such as 16. Oftentimes, these arrays are interconnected in a mannerso that they fit a predefined algorithmic dataflow, which requiresspecifying priority and providing individualized instruction sequences.They are architected based on the reduced instruction set computer(RISC) model, with a reduced set of short instructions. Finally, inthese architectures, each PE is designed to process a single element ofa (wide) data vector at a time (thus the term of vector overlays).

To date, all of these implementations have remained inadequate for manychallenging applications such as real-time, large-format videoprocessing due to limitations in processing and I/O performanceresulting from the array architecture and the addition ofprogrammability. Hence, none of these implementations have achievedwidespread commercial success. Accordingly, improved techniques forproviding efficient signal processing are desired.

SUMMARY

The following presents a simplified summary of the disclosure in orderto provide a basic understanding of certain embodiments of theinvention. This summary is not an extensive overview of the disclosureand it does not identify key/critical elements of the invention ordelineate the scope of the invention. Its sole purpose is to presentsome concepts disclosed herein in a simplified form as a prelude to themore detailed description that is presented later.

Various embodiments of the present invention relate generally todevices, systems, and methods for the performance of digital signalprocessing. According to particular embodiments, a data processingsystem may include a plurality of processing elements. Each processingelement may be an internal hardened field-programmable gate arraydigital signal processor or an internal application-specific integratedcircuit digital signal processor. Each processing element may alsoinclude an internal hardened field-programmable gate array memoryelement, application-specific integrated circuit memory element, orother memory element. The data processing system may also include aplurality of connections between the processing elements, where eachdigital signal processor can directly access the memory element of atleast one other digital signal processor.

In some embodiments, each of the processing elements is configured toexecute one or more instructions received from a designated source basedon an internal state associated with the processing element.Alternately, or additionally, each of the processing elements isconfigured to receive instructions broadcast from a designated source.

In some implementations, the processing elements are arranged in aone-dimensional or two-dimensional grid such that each processingelement is contiguous with one or more other processing elements. Eachprocessing element in the grid can directly access the memory element ofassociated contiguous processing elements. In particular embodiments,data stored in each memory element can be accessed in a single clockcycle.

In some embodiments, each processing element includes an arithmeticlogic unit configured to execute arithmetic and bitwise logicaloperators. The data processing system may be configured to process inputdata arranged in an array, and each processing element may receive arespective sub-array of the input data.

In particular embodiments, the processing elements are arranged in agrid having a first dimension and a second dimension, where the lengthof the grid along the second dimension is greater than the length of thegrid along the first dimension, and where the processing elements areconfigured to receive input data along the length of the seconddimension in order to minimize or eliminate redundant processing. Insome configurations, the length of the first dimension is one processingelement. Additionally, data input and output to the local register fileis implemented as a ring buffer. The processing elements may be arrangedso as to eliminate redundancy in processing data along the first and/orsecond dimension.

According to various embodiments, the processing elements may bearranged in a grid having a first dimension and a second dimension. Insuch embodiments, a length of the grid along the second dimension may begreater than a length of the grid along the first dimension. Theprocessing elements may be configured to receive input data along thelength of the second dimension in order to minimize or eliminateredundant processing. The data processing system may be configured toprocess input data arranged in an array, and each processing element maybe configured to receive a respective sub-array of the input data. Theprocessing elements may be arranged so as to eliminate redundancy inprocessing data along the second dimension.

According to various embodiments, the processing elements are arrangedin a grid having a first dimension and a second dimension. In suchembodiments, a length of the grid along the second dimension may begreater than a length of the grid along the first dimension. Theprocessing elements may be configured to receive input data along thelength of the second dimension in order to minimize or eliminateredundant processing. Data input and output t to the local register filemay be implemented as a ring buffer, and the processing elements may bearranged so as to eliminate redundancy in processing data along thefirst dimension.

In particular embodiments, each of the processing elements is configuredto execute one or more instructions from a designated source, and theexecution of the instructions employs data included in the instructions.

These and other embodiments are described further below with referenceto the figures.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure may best be understood by reference to the followingdescription taken in conjunction with the accompanying drawings, whichillustrate particular embodiments of the present invention.

FIG. 1 illustrates a processing element that includes a single hardeneddigital signal processing core, configured in accordance with one ormore embodiments.

FIG. 2 illustrates a processing element that includes two hardeneddigital signal processing cores, configured in accordance with one ormore embodiments.

FIG. 3 illustrates an example of a configuration of a processing elementinterface, arranged in accordance with one or more embodiments.

FIG. 4 illustrates a single dimensional array of processing elements,which may be referred to herein as a row of processing elements(“ROPE”), configured in accordance with one or more embodiments.

FIG. 5 illustrates a bi-dimensional array of processing elementsconfigured in accordance with one or more embodiments.

FIG. 6 illustrates the general architecture of a signal processor,configured in accordance with one or more embodiments.

FIGS. 7A and 7B illustrate examples of the external I/O logic of theprocessing array for global data insertion and extraction, configured inaccordance with one or more embodiments.

FIGS. 8A and 8B illustrate examples of the external I/O logic of theprocessing array for local data extraction, configured in accordancewith one or more embodiments.

FIG. 9 illustrates an example of an external I/O logic architecture thatcan support both local and global data exchanges, configured inaccordance with one or more embodiments.

FIGS. 10A, 10B, and 10C illustrate examples of image data flow throughthe SIMT processor, arranged in accordance with one or more embodiments.

FIGS. 11A and 11B illustrate examples of the concept of overlappedtiles.

FIGS. 12A and 12B illustrate examples of the mechanism of theconvolution operator and the incurred edge effects.

FIG. 13 illustrates an example of a system that may be used inconjunction with techniques and mechanisms described herein.

DESCRIPTION OF EXAMPLE EMBODIMENTS

Reference will now be made in detail to some specific examples of theinvention including the best modes contemplated by the inventors forcarrying out the invention. Examples of these specific embodiments areillustrated in the accompanying drawings. While the invention isdescribed in conjunction with these specific embodiments, it will beunderstood that it is not intended to limit the invention to thedescribed embodiments. On the contrary, it is intended to coveralternatives, modifications, and equivalents as may be included withinthe spirit and scope of the invention as defined by the appended claims.

For example, the techniques of the present invention will be describedin the context of particular architectures and methods. However, itshould be noted that the techniques of the present invention apply to awide variety of different techniques and mechanisms. In the followingdescription, numerous specific details are set forth in order to providea thorough understanding of the present invention. Particular exampleembodiments of the present invention may be implemented without some orall of these specific details. In other instances, well known processoperations have not been described in detail in order not tounnecessarily obscure the present invention.

Various techniques and mechanisms of the present invention willsometimes be described in singular form for clarity. However, it shouldbe noted that some embodiments include multiple iterations of atechnique or multiple instantiations of a mechanism unless notedotherwise. For example, a system uses a processor in a variety ofcontexts. However, it will be appreciated that a system can use multipleprocessors while remaining within the scope of the present inventionunless otherwise noted. Furthermore, the techniques and mechanisms ofthe present invention will sometimes describe a connection between twoentities. It should be noted that a connection between two entities doesnot necessarily mean a direct, unimpeded connection, as a variety ofother entities may reside between the two entities. For example, aprocessor may be connected to memory, but it will be appreciated that avariety of bridges and controllers may reside between the processor andmemory. Consequently, a connection does not necessarily mean a direct,unimpeded connection unless otherwise noted.

Overview

The present application describes a novel general-purpose signalprocessor that utilizes a combination of programmable logic anddedicated logic such as memory blocks and Digital Signal Processing(DSP) blocks available in certain Field Programmable Gate Arrays (FPGAs)devices. Techniques and mechanisms described herein are also applicableto application specific integrated circuits (ASICs).

Example Embodiments

According to various embodiments, techniques and mechanisms describedherein include a new, high level language programmable processingarchitecture with significant input/output and computational advantageswithout the limitations of conventional FPGA overlays. In someembodiments, this architecture includes arrays of hardened DSP cores andregister files interspersed with generic logic elements (LEs) andinterconnected in a single or bi-dimensional lattice. The processor alsoincludes auxiliary logic modules, which include components such as aninstruction sequencer, an external I/O controller module, mask registersfor conditional execution, and LEs for data pathway selection.

In some embodiments, techniques and mechanisms described herein alsoinclude a highly efficient, FPGA-based signal processing overlay thataddresses limitations of conventional vector overlay approaches. Thisnew architecture may be referred to herein as a tile overlay, while aprocessor that reflects this architecture may be referred to as a tileoverlay processor (TOP). The TOP differs from the vector overlay byarranging the data such that a one-dimensional or bi-dimensionalsub-array of data objects, pixels for instance, is assigned to each PE.This is in contrast to the traditional 2D vector overlay approach ofassigning a single data object to each PE and processing array-sizedblocks of data into the array sequentially. Tile sizes may beapplication and algorithm specific and may be tailored to the size ofthe array, the size of the operator, and the size of the data set. Thetile size may be changed dynamically to accommodate processing data setsizes, for instance high definition (HD), 4 k, and 8 k video or imagedata with the same array.

According to various embodiments, one or more techniques and mechanismsdescribed herein involve aligning the dimensions of an array of PEs tothe size of at least one dimension of the data set, such as, forexample, the width of a video line. The one-dimensional embodiment of aPE array may be referred to herein as a Row of PEs (ROPE). In adifferent embodiment, suitable for applications such as verticallyoriented operations, the PE array may be arranged in avertically-adjacent fashion, which may be referred to herein as a Columnof PEs (COPE). In yet another embodiment, a ROPE/COPE structure(hereafter called ROPE for simplicity) may contain a plurality ofrows/columns of data given available FPGA resources.

In some implementations, once synthesized, the architecture describedherein can execute instruction sequences produced by a compiler or codegenerator from a high-level language such as C. Techniques andmechanisms described herein allow developers to deploy highly complexalgorithms on FPGA devices with a turnaround time comparable to that ofsoftware implementations and performance comparable to that of hardwaredescription language (HDL) implementations.

According to various embodiments, any or all of the techniques andmechanisms described herein may be implemented via ASICs instead of, orin conjunction with, FPGAs. In particular, the techniques that may beimplemented via ASICs may include, but are not limited to: the TOParchitecture, the ROPE architecture, the COPE architecture, the“Line-in, Line-out” architecture, and any architecture in which customDSPs are capable of accessing the memory of neighboring DSPs.

Overlap Penalty

Conventional vector overlay techniques typically impose the drawback ofan overlap penalty. That is, vector overlay arrays of PEs which aresmall relative to the size of the data object, for instance a videoimage, are inefficient when performing neighborhood operations such asconvolutions because data must be duplicated around the edge of thearray-sized subframe to prevent edge effects. A subframe is herebydefined as a contiguous plurality of pixels loaded into the array forconcurrent processing. This inefficiency can increase the processingtime by 2× to 10× for large data sets such as 4 k images.

The overlap penalty is incurred when a large, contiguous data block suchas an image is distributed for processing among a plurality ofprocessing units (array of PEs), so that the array is allocatedcontiguous subsets (or subframes, in the case of images) of the originalblock, one at a time, until the entire data block is processed. Forcertain operators where a resulting datum (e.g., a pixel) is obtained bycombining a contiguous plurality of the original data (e.g., pixels),artifacts called edge-effects are produced at the borders between thesubframes.

FIG. 12A illustrates an example of such operator, extremely common inimage processing, called a convolution. When executing a convolution, aneighborhood of pixels from the original subframe (1200) are multipliedwith coefficients from a kernel (1202) and the results summed up toobtain a single pixel (1203) in the processed subframe (1201). Themechanism illustrated in FIG. 12 (a) is repeated for every pixel of theprocessed subframe. Edge effects are occurring when the processed pixelis close to the border of the subframe, as illustrated in FIG. 12 (b).The projection of the kernel onto the original subframe will now extendbeyond the edge of the subframe and include a neighborhood of pixels(1204) that actually belong to the adjacent (subsequent) subframe (1205)and are not currently available to the array of processors.

In order to ensure the availability of these pixels to the processingunit assigned to subframe (1200), a certain overlap between subframes(1200) and (1205) must be considered. This overlap includes the adjacentbands (1206), which both have a width equal to the half the width of thelargest operator in the process. A more detailed description of aprocedure used to calculate the necessary number of overlapping pixelsis provided in the section dedicated to the Image Data Flow. An overlapbetween adjacent subframes leads to inefficiencies, as the pixels in theoverlapped areas must be replicated, and processed, in both subframes.When the operators are large relative to the array size this can addsignificant overhead, dramatically reducing the efficiency of the array.

According to various embodiments, techniques and mechanisms describedherein such as the TOP overcome this inefficiency. In a TOP, thesubframe size associated to each ROPE may be specified to match thewidth of the data set. More specifically, to process an image of widthZ, one embodiment will comprise a certain number of PEs (n), each with acertain tile width (w), such that n×w=W. In addition, each PE can accesseach of its neighbors' register file, which eliminates the need forredundant data in each PE's register file. These adaptations, combinedwith the “Line-in, Line-out” data flow model described herein,effectively eliminate the need for array subframe overlap in anydimension and dramatically increase the performance of the overlay. Forexample, the performance of the overlay may be increased by a factor of2, 10, or more relative to conventional techniques and depending onfactors such as the operator size and array size relative to the dataset size. A TOP approach also allows very small arrays to operate onvery large data sets. For example, in some embodiments a small,inexpensive FPGA may perform significant processing on a 60 frame persecond, 4 k or 8 k image stream. Such performance had not been possiblewith prior overlays.

Shift Penalty

Conventional vector overlay techniques typically impose the drawback ofa shift penalty. At any given time, a PE in a conventional vectoroverlay has access to a very small number of neighboring data, such asthe four nearest neighbor pixels. As a consequence, for topologicallylarge operators, such as large convolutions, the array must execute amultitude of data transfers (shifts) between adjacent PEs, whichincrease significantly the overall processing time.

According to various embodiments, techniques and mechanisms describedherein such as the TOP reduce or eliminate the shift penalty since eachPE contains all of the neighboring data objects it needs per the tiledefinition. In addition, in embodiments where each PE has access to itseast and west neighbor memories, the reach of each PE is tripled. Eachsubsequent neighbor connected to the PE multiplies the data reach by thesize of the tile reducing or eliminating the need for costly data shiftsin the array. For instance, in some embodiments a PE may have immediateaccess to 256×256 or more data elements.

Integer Normalization Penalty

Conventional vector overlay techniques typically impose the drawback ofan integer normalization penalty. Vector overlays often include integermultipliers, adders, and arithmetic-logic units (ALUs) and,consequently, can only operate in fixed-point arithmetic. In fixed-pointarithmetic, a multiplicative scale factor must be introduced to mitigateaccuracy limitations, especially when small numbers are involved. Thisscale factor must then be factored out of the result, in an operationcalled normalization, which adds inefficiency in processing.

According to various embodiments, techniques and mechanisms describedherein such as the TOP can avoid normalization through costly shiftregisters or additional data passes through the PE by custom pre-scalingthe coefficients in the instruction word. These advantages may beprovided even when the TOP is implemented as an integer processingmachine that operates in fixed-point arithmetic. The scale factor may bechosen so that the most significant bit of the output data is aligned toa predefined bit in the output port of the PE. In this manner, thenormalization of the result is executed by discarding the appropriatenumber of least significant bits in the output bus. Such an approach maybe implemented at zero hardware cost and may enable scaling to verylarge arrays.

Input/Output Inefficiency

Conventional vector overlay techniques typically impose the drawback ofI/O Inefficiency. In most parallel processors, the required datatransfers (I/O) from the external memory into the internal/local memoryis more time consuming than the computational process itself. In theseprocessors, the challenge is in being able to execute the I/O within therequired time constraints, rather than in the data processing. Suchprocessors may be referred to as I/O limited, as opposed to computelimited.

According to various embodiments, techniques and mechanisms describedherein such as the TOP are optimized for I/O in a multitude of aspects.First, the “Line-in, Line-out” approach may store the I/O data in theregister file configured as a ring buffer, eliminating vertical overlapand eliminating redundant I/O. In addition, this approach may load onlythe data needed to begin the operation rather than all of the datarequired for the entire operation. The ROPE approach may reduce oreliminate processing overlap, which avoids processing of replicatedpixels. This reduces or eliminates I/O redundancy, which can limit theoverall performance of the processor. Second, each PE has access to itsneighbors' memory, reducing or eliminating the need for data movementbetween PEs. Third, data are efficiently inserted into or extracted fromthe array in either a local or global manner with minimal overhead, asdescribed in a later paragraph. Finally, the instruction sequencer isdesigned to flag the particular instructions during which the pipelinethrough the PE is stalled, or which comprise cycles with no memoryaccess. During these cycles, the instruction sequencer may scheduleexternal I/O operations in preparation of future processing. Theseexternal I/O operations, scheduled while the processing is still ongoingin the PE, are called concurrent I/O and may contribute to a significantincrease of processing efficiency, especially in processors orapplications that are I/O limited.

Processing Element (PE)

General Description

According to various embodiments, the architecture may include a singleor a bi-dimensional array of Processing Elements (PEs). Each of the PEscomprise one or more hardened DSP cores, or slices, such as the DSP48E1and the DSP48E2 available in Xilinx Ultrascale and Ultrascale+ devices.Each PE is connected to a high speed register file, constructed frominternal memory blocks, often called Block RAM or Ultra RAM. Forexample, in certain Ultrascale and Ultrascale+ devices, this memoryconsists of blocks of 4,096 72-bit words, each of which can be dividedinto 18-bit wide segments to support 4 PE's. The invention is notlimited to Xilinx devices and can be applied to, for example,Altera/Intel devices with DSP IP cores. One example of such a device isArria 10, where a floating-point DSP core is available.

In particular embodiments, the term register file may refer to a type ofmemory accessible in a single clock cycle. In some implementations, adifferent type of memory block may be employed in place of a registerfile, such as a memory block accessible in two or more cycles.

A single-core PE embodiment (102) is illustrated in FIG. 1. In a singledimensional array, a PE, labeled, for example, PE k, may access threeregister files: its own, labeled k (111), and those of its immediateneighbors, labeled k−1 (110) and k+1 (112), respectively. Auxiliarylogic (103, 104) is added to assist in selecting the data pathways inand out of the PE, The PE instruction sequencer (101) isolates theconstants from the instruction stream and delivers them to each of thePE inputs. The embodiment shown in FIG. 1 includes a fork mask register(105) that masks data outputs for conditional execution (forks), whichdifferentiates the SIMT architecture shown in FIG. 1 from a conventionalSIMD architecture.

In some implementations, the PEs may be arranged in a bi-dimensionalarray, and the connectivities among neighboring PEs may change relativeto a single-dimensional array implementation. For example, one PE mayaccess the register files of its east, west, north, south, and diagonalneighbors, besides its own register file. The techniques and mechanismsdescribed herein covers both topologies (single-dimensional andbi-dimensional arrays), which are discussed herein.

In some embodiments, a PE may comprise two DSP cores, as illustrated inFIG. 2. In the embodiment shown in FIG. 2, the input and outputselection logic (200, 205) is still present. The two DSP cores (201 and202) are separated by selection logic (203, 204), which allows for thedynamic selection of the first DSP core (201) as a standalone unit, orof both DSP cores in cascade. This configuration allows for a greaterflexibility and faster speed. It also alleviates the I/O traffic betweenthe PE and local register files, but, on the other hand, it reduces thetotal possible number of available PEs by 50% and, consequently, thesize of the SIMT array. The decision to use single or dual core PEsdepends on factors that may include, but are not limited to: the FPGAdevice family, the configuration of the available internal memory, thetype of operators that must be supported, and the necessary executionspeed. For example, a Xilinx DSP48E2 core can execute aMultiply-Accumulate (MAC) operation in one clock, but not an AbsoluteDifference (AD). The addition of a second DSP48E2 core would ensure thatan AD can also execute in a single clock. On the other hand, on anAltera/lntel Arria 10 native fixed point DSP core, an AD does execute ina single clock.

In some embodiments, a PE may include three or more DSPs. For example,such a configuration may be desirable when performing calculations ofincreased complexity. In typical FPGA devices, DSP slices are cascadablethrough high speed data buses which may or may not be used inconfiguring a PE. In general, the techniques and mechanisms describedherein are generally applicable to a variety of complex PEarchitectures, such as those including three or more PEs.

Internal PE Architecture

The block diagrams illustrated in FIGS. 1 and 2 show a ProcessingElement (PE) that includes one or two hardened DSP slices and a registerfile configured from block RAM or Ultra RAM memory generally availableon most FPGA devices. In FIG. 3, a more detailed representation of a PEis provided in accordance with one or more embodiments. In particular,FIG. 3 illustrates internal connections that may exist within the PE.

The central part of the PE is the DSP core (300), which may include one,two, or more hardened DSP cores. Detailed in FIG. 3 is a breakdown ofthe connections attached to the multiple inputs that may be available ina DSP core. The 4 input ports shown in FIG. 3, namely, A_IN, B_IN, C_IN,and D_IN, respectively, are present in a Xilinx DSP48E DSP slice, butthe techniques and mechanisms described herein may be applied to anysuitable DSP cores. For example, the same design could apply to anAltera/Intel Arria 10 device, where a number of up to 6 different inputports, namely dataa_x0, dataa_v0, dataa_z0, and datab_x1, datab_y1, anddatab_z1 are available, depending on the configuration. The PE alsoincludes a Register File (301).

In particular embodiments, the techniques and mechanisms shown in FIG. 3are applicable to a structure called a row of PEs (ROPE), which is asingle-dimensional SIMT array where every PE can access its own registerfile, as well as the register files of two neighbor PEs, referred to as“East” and “West.” These connections are available on ports A and D,respectively, through multiplexers A Mux (303) and D Mux (304),respectively. In this embodiment, ports A and D can also receive inputfrom local memory ports A_OUT and B_OUT, respectively. In addition, aconstant that may be retrieved directly from the instruction by aninstruction parser and decoder module is transmitted to port B_INthrough input multiplexer B_MUX (302). The remaining input port, C_IN,is connected to another constant from the instruction stream via theC_MUX (305), or to the DSP output, P_OUT, to provide a feedback loop ina multi-pass operator.

The connections shown in FIG. 3 have been configured in such a way so asto facilitate the basic operator of convolution with symmetriccoefficients. In one configuration, ports A and D are inputs into apre-adder block, whereas port B leads directly to the multiplierimmediately downstream from the pre-adder. Two independent pixels may beloaded from the memory concurrently on A_IN and D_IN, then addedtogether in the pre-adder, and finally, multiplied with the filtercoefficient B_IN they both share through symmetry. Using this strategy,two multiply/accumulate (MAC) operators may be executed in a singleclock for a symmetric filter, Such a structure may be employed in Xilinxdevices, Altera/Intel devices, or other suitable DSP cores.

A PE can be configured in a different manner than the one shown in FIG.3. For example, other configurations may be employed depending onfactors such as the application at hand and the available resources onthe FPGA device.

In the embodiment in FIG. 3, the I/O_IN (310) and I/O)_OUT (311) lineshave been explicitly represented to show an embodiment of the externalI/O logic attached to each PE. The external I/O logic creates theinterface between an I/O Controller and the local register files, andserves as a platform for data transfer between an external memory, ordata stream, and the local PE memories. The present invention does not,in any way, require that the external I/O be supported by the A_IN portsand A_OUT ports, exclusively. A port allocation different from theexamples discussed here is well within the scope of the techniques andmechanisms described herein.

Also represented in FIG. 3 is an embodiment of the mechanism throughwhich the PE stores a result into the register file. In this embodiment,the multiplexer SRC_MUX (307) selects the source of the datum to bestored, which may come from the DSP output (P_OUT), or another registerfile output such as A_OUT.

If the datum to be stored comes from a register file output, the PE issaid to be capable of direct memory loopback. Through a direct memoryloopback, the PE can efficiently move a datum from one register filelocation to another without having to pipeline it through the DSP core.In the embodiment shown in FIG. 3, the datum is read through registerfile port A_OUT and transmitted through SRC_MUX to the B_IN port. Thedirect memory loopback facilitates increased efficiency because it maybe conducted concurrently with other operators that execute on the DSPcore. In different PE embodiments, other ports may be used to implementthe direct memory loopback capability.

If the datum to be stored into the register file comes from the DSPoutput (P_OUT), then one must contend with an important limitation ofthe typical DSP cores available in FPGA devices. Specifically, the widthof the local memory words in both Xilinx and Altera/Intel FPGA devicesis typically programmable, but is limited in comparison to the commonoutput width of an embedded DSP core. For example, a block RAM module ina Xilinx device is 18-bits wide, but a Xilinx DSP slice can outputresults with 48 bits of precision. An M20K memory block in anAltera/Intel device is typically 20-bits wide, but the Altera/Intel DSPcore can provide 64 bits of precision. In order to circumvent thismismatch and increase the computational precision of the PE array, amulti-cycle register file access is implemented in the PE. In someembodiments, the multi-cycle access may be implemented by splitting theoutput line in two equal widths and feeding them into separate inputs (2and 3, in FIG. 3) of the SRC_MUX (307). The memory store operation willbe performed in two cycles, one per each multiplexer input.

Another meaningful capability of the PE made possible by embodimentshown in FIG. 3 is the indirect addressing capability. In the embodimentfrom FIG. 3, the address for register file port B may be provided eitherexplicitly, in the instruction (309), or by the DSP unit itself, as aresult of a prior computation. This latter scenario may be selectedthrough the indirection bit (308), which is also provided in theinstruction. A common application that will be significantly expeditedby indirect addressing is a Look-Up-Table (LUT). A LUT is an example ofcontent addressable memory where every input is mapped into a differentoutput based on a transformation function defined synthetically,point-by-point. In image processing, many enhancement techniques, suchas, for example, fixed and adaptive contrast and brightnessenhancements, may be implemented through a LUT. In order to implement aLUT, a certain area in the memory will be allocated for storing the LUTtransformation. Then, the DSP core will compute the input value into theLUT and send it to ADDR_B port of the local register file through theSRC_MUX and ADDR_MUX (306). The selection bit of ADDR_MUX will be set to1 by the INDIRECTION (308) bit provided by the instruction.

It is important to emphasize that most DSP cores in an FPGA device aredesigned to be interfaced efficiently with one another, in order tosupport processing of wider data elements or cascading operations. Theinterface between two DSP cores described herein may, or may not use theinterface signals provided by the FPGA manufacturer.

Single Instruction-Multiple Thread (SIMT) Array of PEs

According to various embodiments, by interconnecting multiple PEs in asingle or bi-dimensional array, a highly parallel processor may besynthesized. A single-dimensional embodiment of such an array isillustrated in FIG. 4, where a number of 2n PEs (401) and register files(400) are chained together in a ROPE. In this embodiment of a ROPE, PEnumber k has access to register files k, k−1, and k+1, as shown inFIG. 1. A ROPE also has the advantage that it can be synthesized withminimal routing because the topology of the architecture follows thephysical column-based layout of the DSP cores and register files in mostFPGA devices. The particular ROPE embodiment depicted in FIG. 4 includesthe wraparound connections (403) and (404), which ensure that arrays ofdata stored in the register files 1 to 2n can undergo rotation withwraparound.

In some embodiments, for applications such as 4 k or 8 k videoprocessing, a bi-dimensional mesh embodiment may be a higher performingarray configuration, particularly if there are a large number of DSPsavailable in the device. FIG. 5 shows a bi-dimensional, m×n PE array. Abi-dimensional array requires additional multiplexer resources and maybe harder to route on the device because each PE nominally has access tofive register files, namely, its own, and those attached to its north,south, east, and west neighbors. Wider multiplexers may be employed toprovide access to neighbors up to n PE's away in either direction, ordiagonally. In both single-dimensional and multi-dimensional meshes,connection resources may be reduced significantly by serializing theconnections between PE's.

According to various embodiments, both single-dimensional andbi-directional PE arrays are suitable architectures for the SingleInstruction Multiple Thread (SIMT) computational model. In this model,each PE executes the same instruction sequence on distinct data residingin its own register file, or in the adjacent register files. In FIG. 6,the global architecture of the signal processor is illustrated. Theinstruction stream (600), which includes information such as opcodes,addresses, and constants, is parsed and decoded by the InstructionParser and Decoder block (601). The resulting signals includeinformation such as of addresses, constants, and control signals. TheI/O instructions are sent to the I/O controller (604), which controlsthe data. I/O between the external and the local SIMT memory. The I/Ocontroller generates the memory control signals for both the externalmemory (603) and the local register file memory that is part of the SIMTarray (602). Local memory addresses and control signals for the SIMTarray may selectively come from either the I/O controller, or theinstruction parser. The selection is accomplished through themultiplexer (605).

External I/O

Global Data Extraction

In some embodiments, the local register files of the PE array must havethe capability to exchange data with an external memory. Downloading anoriginal image into the local register files for processing, anduploading the resulted image back into the external memory are twoexamples of such data exchange operations. FIG. 7A depicts the I/O logicarchitecture that executes a global data output operation from a ROPEcomprising the PEs numbered 0 to n−1. In a global data output operation,all of the PEs in the ROPE will sequentially send their output data tothe I/O controller (700). In this embodiment, each of the PEs' localregister file RAM block (701) is connected through its local memory portA with a multiplexer (702) followed by a pipeline register (703). Whenthe I/O controller (700) is ready to collect the data from the ROPE, itwill switch all multiplexers (702) to input 1, which allows the pipelineregisters (703) to load from the PE local register file memory ports A.In the next step, the multiplexers are switched to input 0, which allowsthe data from each PE to propagate from register to register towards thecontroller. The presence of the pipeline registers (703) is indicatedfor two reasons. First, during the n clocks required by the I/Ocontroller to collect all the data from the local register files of theROPE, the memory ports A will be freed for other access. Secondly, thearchitecture will be able to support long ROPEs because the datatransfer to the controller is pipelined. The I/O controller can servicemultiple ROPEs concurrently.

Global Data Insertion

For global data insertion into the local register file memories of aROPE, a similar pipelined architecture is described in FIG. 7B. In theembodiment shown in FIG. 7B, the multiplexers (702) are replaced withdemultiplexers (704). A new set of registers (705) are included in thepipeline. The local register file memory port for data insertion is nowPort B. If the local register file memory allows it, the insertion andextraction can be executed concurrently, since they are using separateports. However, this invention does not require that separate memoryports be used. For global data insertion, the data are being pushedthrough the pipeline by the I/O controller until all the pipelineregisters are loaded. For this step, the demultiplexers are set tooutput 0. When all the pipeline registers are loaded, the demultiplexersare switched to output 1, which allows the register file memory ports toaccess the data from the pipeline registers.

Local Data Extraction

In some embodiments, there is a need in some applications for local dataextraction and insertion, where only selected PEs exchange one of morepieces of data with the external memory through the I/O controller. Anexample of such application is a global image minimum. To calculate theminimum pixel in an entire image, all PEs will first determine theirrespective local minima, which are the minimum pixels from thepartitions of the image that are stored in their local register files.Since neighboring PEs can access each other's register files, theselocal minima may then be propagated to adjacent PEs and pairwisecompared against other, adjacent local minima. Following this strategy,after an appropriate number of steps, the global image minimum will beavailable in only one of the PEs.

According to various embodiments, in order to retrieve this globalminimum, the I/O controller will collect data from all PEs, even thougha single piece of datum, from a single PE, is of interest. In FIG. 8A, apipeline architecture is shown that allows the I/O controller to polland extract data from either a single, or a set of specific PEs, whilebypassing the rest. One advantage of this I/O logic embodiment is thatthe PEs themselves may signal if their register files have dataavailable for extraction. The I/O controller need not have priorknowledge about the locations of the data to be extracted.

In the embodiment shown in FIG. 8A, the local data extraction operationis accomplished by employing a custom designed “gate” (804), hereuponreferred to as the Gate. One embodiment of the Gate is detailed in FIG.8B.

The local data extraction architecture in the embodiment shown in FIG.8B functions as follows. When a PE has data available for extraction, itloads it into its output register (or registers) R (802) and raises alogical 1 on its M line. For example, a PE based on a Xilinx DSP core(DSP48E1 or DSP48E2), may employ the PATTERNDETECT output for the Mflag. In other devices, the flag M may be a custom 1-bit output, or itmay employ the MSB, or any available bit, from the data output. Theinvention is not limited to these examples. Instead, techniques andmechanisms described herein may be applied to, for example, anyimplementation where a PE uses an output to signal that it has dataavailable for extraction. In steady state mode, the IO_ENABLE (805)signal is set to 0, and all the Gate circuits all output 1, keepingcontinuity on the DATA_OUT bus (807). When the I/O controller (800) isready for data extraction, it will raise the IO_ENABLE line to 1. If PEdoes not have data available for extraction, its line M will stay on 0,and the element labeled Gate 0 (804) will output 1, maintainingcontinuity on DATA_OUT bus (807). If, on the other hand, PE0 (801) doeshave data available for extraction, it will have its M line raised to 1.Consequently, Gate 0 will have logical 1's on all of its inputs, namelyA, B and C. This combination of inputs will cause its output to drop to0, which switches the multiplexer (803) to input 0, thereby allowing theI/O controller access to its output register R0. On the next clockcycle, Gate 1 (the next Gate element in the pipeline) will see its inputB raise to 1 and will start a similar process: If the M line is raisedin PE 1, then MUX 1 is switched to input 0 and connect the register R1to the DATA_OUT bus (1307). If, on the other hand, the M line is low,which means that PE 1 does not have data available for extraction, thenMUX 1 stays on its input 1, maintaining continuity of the DATA_OUT bus(807).

In some embodiments, the output register (802) may be a plurality ofregisters which for example might also hold the row and column number ofthe PE providing the data. The output register may be loaded by the PEor by a discrete row/column position register.

Following this sequence for every PE, the gated data extractionarchitecture illustrated in FIG. 8B will execute a complete sequentialdata extraction over the entire ROPE. When the cycle is complete, theoutput signal from the last Gate device in the ROPE will drop its outputto 0 (806), thereby communicating the I/O controller that the cycle hasended (SCAN_COMPLETE) and that all of the available data have beenuploaded. At this time, the I/O controller may switch the IO_ENABLE lineto 0.

To summarize, the local data extraction architecture shown in FIG. 8A isstructured around 4 distinct data and control buses: the DATA OUT bus(807), the BUS_AVAILABLE line (808), the IO_ENABLE line (805), and theSCAN_COMPLETE line (806). The IO_ENABLE control line signals that theI/O controller is ready for local data extraction. The SCAN_COMPLETEline is an extension of the BUS_AVAILABLE line, which signals back tothe I/O controller that the local data extraction cycle has beencompleted. The DATA_OUT bus spans a number of multiplexers, out ofwhich, only one (for example, MUX k) is switched to 0 at any givenmoment, allowing access for the I/O controller to the local dataregister Rk. It is important that all the multiplexers from MUX k−1 toMUX 0, be switched to 1 at this time, in order to ensure the continuityof the DATA_OUT bus, from the I/O controller up to MUX k. Finally, theBUS_AVAILABLE control line goes through the Gate devices and signals toany Gate (for example, Gate k) that the DATA_BUS is available forextracting data from register Rk.

In some embodiments, in order to support the sequence of operationsdescribed above, the Gate element implements the following behavior.After a specific Gate (for example, Gate k) has granted access to theI/O controller to its data register (Rk) by dropping its output to 0, itmust automatically switch back to 1 in the next clock cycle, in order torestore the continuity of the DATA_OUT bus (807) so that the rest of theROPE can continue the data extraction process. One embodiment of acircuit designed to support this functionality is the Gate element shownin FIG. 8B. The embodiment from FIG. 8B is not unique, and anyalternative circuit that implements the sequential, selective dataextraction described above may be employed. The Gate element (809) isenclosed within the hashed rectangle and has three components, namely, asynchronous latch element (810), and two AND gates (811) and (812). WhenBUS_AVAILABLE_IN, input C (from the M output of the PE) and IO_ENABLEare all on a logical 1, the output of the Gate is changed to 0, therebyswitching MUX k to 0. This interrupts the DATA_OUT bus. On the otherhand, the output of the internal AND gate (812) is also switched to 1,which, on the next clock, will reset the latch. This will change theoutput of the Gate to 1, thereby restoring the continuity of theDATA_BUS bus and enabling the next Gate (k+1) to start the process.

Local Data Insertion

In some implementations, a similar circuit may be employed for localdata insertion, an operation that is complementary to the local dataextraction discussed in the section above. In local data insertion, anindividual PE may extract a piece of data from the I/O controller, inorder to download a sequence of data into the array in a data dependentmanner. The logic for local data insertion may be derived from thatpresented in FIG. 8A for local data extraction in a similar manner tothat used to derive the global data insertion (FIG. 7B) from the globaldata extraction (FIG. 7A). When resources permit, different memory portsmay be available for the data extraction and insertion, in order toallow for concurrent memory read and write operations.

Local and Global Data I/O

According to various embodiments, the I/O logic architectures for global(FIG. 7) and local (FIG. 8) data exchanges may be combined, for instancein order to support both types of operations. The building block of thishybrid I/O logic is shown in FIG. 9, in an embodiment for local andglobal data extraction. The embodiment shown in FIG. 9 includes a seriesof data exchanges for data retrieval. For example, the data exchange forRAM 0 (900) includes two multiplexers, MUX 01 (901) and MUX 02 (902),which encapsulate the output data register R0 (903). For global dataextraction, MUX 02 (902) is set to output 1, while MUX 01 (901) willswitch from 1 to 0. While MUX 01 (901) is on 1, the register R0 isloaded from the local memory. Then, MUX 01 (901) switches to 0, whichallow the data to travel, from register to register, to the I/Ocontroller. For local data extraction, MUX 01 (901) is always set to 1,while MUX 02 (902) will be driven through the Gate element presented inthe previous section (not represented in FIG. 9 for the sake ofclarity). This pattern of data access continues across RAM 1 through RAMN.

In the embodiments described for global and local data I/O, thisinvention does not require use of specific register file ports. Port Aand B may be assigned differently in various embodiments. Assigningdifferent ports for data input and output maximizes the I/O performancewhile assigning the same ports maximizes the PE's performance as thesecond port is always available to the PIE.

Image Data Flow

According to various embodiments, due to its massive parallelism, theSIMT array described herein is particularly suitable for real time videoprocessing, especially for large image formats such as Ultra HighDefinition (UHD), and 8 k. For the single dimensional array (ROPE) shownin FIG. 4 and the bi-dimensional array shown in FIG. 5, several imagedata flow patterns are possible. Which data flow pattern is mostappropriate depends on factors that may include, but are not limited to:the number of PEs available on the device, the size of the localregister files, and the nature of the application.

According to various embodiments, FIG. 10A illustrates an example of animage data flow pattern that is suitable to a SIMT array in the shape ofa row of pixels (ROPE). Assuming that the ROPE comprises a number of nPEs, and that a total number of m×n PEs are available in the device,each of the m ROPEs can be assigned a band in the image (1000). For animage of W×H pixels, the width of the band of pixels processed by theROPE will be W pixels and the height of the band of pixels processed bythe ROPE will be

$\frac{H + {m \times {overlap}}}{m}.$

This is also called the tile height, or sub-array height.

Given an available number of PEs in a device, the number of ROPEs m maybe determined by first selecting the horizontal tile size. This conceptis represented in FIG. 10B. A tile (1003), or sub-array, is horizontallysized so that all of the pixels required for the largest operator can bestored in the PE's register file or in that if its connected neighbors.The number of PEs per Row n is then the image width W divided by thetile width. Subsequently, the number of ROPEs m is the number ofavailable PE's divided by the number of PE's per row ii. In thisexample, adjacent tiles do not require overlap because each of the PEshas access to the register files of its neighbors. However, the zeropadded regions (1011) along the long edges of the image are stillincluded, even if the lateral PEs are connected to each other throughROPE wraparound (see FIG. 4).

Although adding the second dimension to the ROPE introduces verticaloverlap, it may be acceptable in order to increase the processing powerof the array with the available resources. According to variousembodiments, the overlap (1002) is number of pixels that the adjacentbands will share. The size of the overlap may be determined based onfactor such as the topological size of the image operators that arebeing applied. For example, if a convolutional operator with a size of5×5 pixels is to be applied, then an overlap of 5−1=4 pixels isnecessary in order to avoid edge effects at the seams between the bands.By a similar rationale, the beginning and the end rows of the image issupplemented with vertical zero padded regions (1010) that are half ashigh as the overlap areas. Horizontal zero paddings (1011) are furtherrequired at the left and right edges of the image. The heights of thevertical zero-padded regions and the widths of the horizontalzero-padded regions may be set to half the height of the overlappedregions.

According to various embodiments, zero padding may be substituted byother types of padding including data mirroring, data repeating, etc.Alternately, no padding at all may be specified, with or without datawrapping to the other edge of the array.

Depending on the application, additional storage space may be maintainedin the local register file sufficient for performing processingoperations. For example, space may be reserved in the local registerfile sufficient to store the entire number of intermediate imagesrequired by the process. Failure to do so will result in less efficientprocessing since it will require additional data transfers between thelocal register file and the external memory.

According to various embodiments, if the memory space in the localregister file is insufficient to store the desired tile height and therequired intermediate images, the data flow may be adjusted to storeonly fragments of the tile at any given time. This creates a largevirtual tile extending tall enough vertically to enable the entireheight of the image to be distributed evenly across the number of ROPEsavailable without requiring the register file to be large enough to holdthe entire tile and intermediate images. This concept is represented inFIG. 10C. Starting from the top of the image tile, just enough lines toallow the topologically largest operator to perform are loaded. Forexample, if this operator is a 5×5 pixel convolution, then 5 lines fromthe tile are loaded. Convolution is then performed on these 5 lines anda one-line result is produced. The next adjacent image line is theninput while the first (oldest) line will be removed. After these steps,the register file will contain the pixels required for the next sequenceof convolution operators, producing the next line of the result. Such aconfiguration is referred to herein as a “Line-in, Line-out” model. Forexample, FIG. 10C includes tile (1006) with a portion of the linesloaded, as bounded by the line in (1004) and the line out (1005). Byenabling the compiler to support symbolic addresses, the register filecan be configured as a ring buffer. In the ring buffer, the newest linewill physically replace the oldest line, which is discarded. The fivelines stored in the register file need not be physically contiguous, butcan appear to be with symbolic addressing. In this manner, the remaininglines need not be shifted in the memory in order to maintain theirphysical order in the image.

FIG. 11A illustrates an embodiment in which image tiles are horizontallyoverlapped. For example, image tile 1 (1100) includes an overlappingarea (1101) that overlaps with image tile 2 (1102). Such a configurationmay be desirable for any of various reasons. For example, if the routingresources on the device are limited, then overlapped tiles wouldalleviate the need to ensure that every PE has access to register filesof its adjacent neighbors, increasing the number of PE's possible in thedevice. Similar to the vertical overlaps described in FIG. 10, the tileoverlap may be configured so as to cover at least half of the largesthorizontal operator. Another reason to employ horizontal tile overlapsis related to the partitioning of the pixels among the array PEs. Whenperforming image processing, one may allocate a horizontal image band toeach ROPE, as described with respect to FIG. 10A. When allocating thenumber of PEs in a ROPE and the total number of ROPEs on the device,relevant criteria may include, but are not limited to: the size of theimage, the size (footprint) of the largest operators, and the totalnumbers of PEs on the device. Allocating too few PEs per ROPE willresult in a large number of ROPEs; if large vertical operators areemployed in processing, this allocation is inefficient as the cost of alarge vertical overlap will be incurred with every ROPE. Increasing thenumber of PEs per ROPE such that a very small number of ROPEs arerequired increases processing efficiency with large vertical operators.The subsequent reduction in tile width can be mitigated by employinghorizontal overlap. If the number of PEs per ROPE is too large for theimage width, then the tiles will be too small horizontally. Even withEast-West neighboring access, the width of the tile may prohibitefficient large horizontal operators.

FIG. 11B illustrates an embodiment in which tiles share pixels. Forexample, the tile number k in FIG. 11B includes pixel blocks (1104),(1106), and (1108). Pixel blocks (1104) and (1106) are shared with tilek−1, while pixel blocks (1106) and (1108) are shared with pixel blockk+1. In this example, if each PE processes an 8 pixel wide area in thetile (the hashed region in each tile), a number of 240 PEs per ROPE willbe allocated for a 1920 pixel wide image, which is quite reasonable.However, an 8-pixel wide area will only support a 24-pixel wideoperator, and only if neighboring access had been implemented. If, onthe other hand, the tiles have a 24-pixel width with a 16 pixel overlapon each side, each PE will be able to support a 40-pixel wide operatorwith neighboring access. Such a configuration may impose an I/O costassociated with traffic between the external memory and the localmemory, as many of the same pixels will be transported to multipledestinations (local blocks).

In some implementations, a compiler automatically determines the best PEallocation for a ROPE as well as parameters such as tile sizes and tileoverlaps in order to maximize the processing efficiency.

In some instances, there may be no overlap of any kind. For example, thetile size may be selected such that the (tile width×number of PE's)exactly matches the width (or height) of the data and the tiledimensions are selected to enable access to all data required for thelargest operation from the local register files or the register files ofthe accessible neighbors. In this example, only a single ROPE isemployed and there is no overlap, allowing processing and I/O to be 100%efficient.

In some embodiments, multiple data channels can be processedconcurrently via a SIMT array to take full advantage of the availablecomputing power. These data channels may include external data storagemodules such as memory banks, or data streams. In some implementations,more than one image stream can be processed by the array at the sametime. For example, a certain equal number of PEs is assigned to eachchannel. In some embodiments, PEs can be distributed to multiple datastreams and allowed to perform distinct operations, concurrently. Suchan array is referred to herein as a multiple instruction-multiple data(MIMD) machine and may include multiple instruction sequencers.

Configuration

FIG. 13 illustrates one example of a computing device. According toparticular embodiments, a system (1300) suitable for implementingparticular embodiments of the present invention includes a processor(1301), a memory (1303), an interface (1311), and a bus (1315) (e.g., aPCI bus or other interconnection fabric).

Particular examples of interfaces supported include baseband interfacessuch as SDI, HDMI, DisplayPort, MIPI and the like; and networkinterfaces such as Ethernet, frame relay, cable, DSL, token ring, andthe like. In addition, various very high-speed interfaces may beprovided such as Infiniband interfaces, 100 Gigabit Ethernet interfaces,ATM interfaces, HSSI interfaces, POS interfaces, FDDI interfaces and thelike. Generally, these interfaces may include ports appropriate forcommunication with the appropriate media. In some cases, they may alsoinclude an independent processor and, in some instances, volatile RAM.The independent processors may control communications-intensive taskssuch as packet switching, media control and management.

Techniques and mechanisms described herein may be used to bringultra-high-performance FPGA development to software developers withlittle or no experience in RTL and hardware synthesis. In this sense,techniques and mechanisms described herein may affect FPGA basedprocessing in a fashion similar to the one the advent of the ComputeUnified Device Architecture (CUDA) platform had on GPU adoption as ageneral-purpose compute engine.

In some implementations, several layers of software are interposedbetween the developer and the FPGA device. In order to assist inenabling the signal processor to become a general purpose computationalengine, the system may include a high-level language compiler orinterpreter, such as the compiler/interpreter (1317) shown in FIG. 13.Programs written for the SIMT engine may be written in C, or Python, oranother such language. A set of SIMT intrinsics captures thefunctionality specific to the SIMT engine and is parsed and interpretedby a library, for example, which translates these intrinsics intomachine code instructions. These machine code instructions may be devicespecific, since an Altera/Intel FPGA device may need a differentinstruction set than a Xilinx counterpart, and an ASIC implementationmay have a fully custom instruction set.

According to various embodiments, each high-level instruction istranslated by an instruction parser and decoder. In the translated code,addresses, constants, and control signals required by the DSP sliceswill be extracted and passed on to an instruction sequencer. Thecompiler/interpreter (1317) may include an optimization layer, whichwill detect sequences of instructions that could be optimized. Forexample, if the computational sequence executed in the DSP slice of thePE requires a pipeline stall, then those cycles may be marked by theoptimizer as candidates for concurrent I/O, or I/O instructions thatexchange data between the local register files to the external memory.

Global Immediate Data

According to various embodiments, instructions broadcast to a PE arraymay include immediate data used in the execution of the instructionswith which they are included and/or in the execution of instructionssent at a later time. For example, the FPGA may include a globalinstruction sequencer with a global immediate data lookup table toimplement indirect immediate data references within a PE arrayinstruction. In this example, the PE array instruction may include anaddress that references a value stored in the immediate data lookuptable to be used by PEs in execution the instruction and/or laterinstructions.

In some implementations, the global immediate data lookup table may bewritten by an external data path to allow one or more externalprocessors to dynamically update the PE array instruction immediate dataglobal indirection value. In this way, feedback of external decisionsmay be provided to influence the processing done by the PE array.Alternately, or additionally, the table may be updated by the PE arrayinstruction stream.

Multiple Array Configuration

According to various embodiments, techniques and mechanisms describedherein may be used in conjunction with multiple array processors (MAPs).MAPs may be arranged in any of various ways. For example, each MAP mayprocess a separate channel (i.e. stream) of data. As another example,each MAP may operate on a distinct segment of an image. As yet anotherexample, MAP processors often have a long edge and a short edge. In thiscase, two or more MAP processors may be arranged such that they areadjacent along the short edge.

In some implementations, flexible signal instruction streams may feedone or more MAPs that are configured to process different data streams.Indirection may be used to allow different global constants per streamfor applications such as compression quantization.

In particular embodiments, an FPGA may include one or more MAPs.Alternately, or additionally, an ASIC may include one or more MAPs. Insome configurations, MAPs may receive instructions from multipleinstruction sequencers.

CONCLUSION

In the foregoing specification, the invention has been described withreference to specific embodiments. However, one of ordinary skill in theart appreciates that various modifications and changes can be madewithout departing from the scope of the invention as set forth in theclaims below. Accordingly, the specification and figures are to beregarded in an illustrative rather than a restrictive sense, and allsuch modifications are intended to be included within the scope ofinvention.

1. A data processing system comprising: a plurality of processingelements, each processing element including a respective internalhardened field-programmable gate array digital signal processor, eachprocessing element further including an internal hardenedfield-programmable gate array memory element; and a plurality ofconnections between the processing elements, wherein each digital signalprocessor can directly access the internal hardened field-programmablegate array memory element of at least one other digital signalprocessor.
 2. The data processing system recited in claim 1, whereineach of the processing elements is configured to execute one or moreinstructions received from a designated source, the execution of the oneor more instructions based on an internal state associated with theprocessing element.
 3. The data processing system recited in claim 1,wherein the plurality of processing elements is arranged in aone-dimensional or two-dimensional grid such that each processingelement is contiguous with one or more other processing elements,wherein each processing element can directly access the internalhardened field-programmable gate array memory element of associatedcontiguous processing elements in a single clock cycle.
 4. The dataprocessing system recited in claim 1, wherein the data processing systemis configured to process input data arranged in an array, and whereineach processing element is configured to receive a respective sub-arrayof the input data.
 5. The data processing system recited in claim 1, thedata processing system further comprising: an input/output subsystemconfigured to write input data to the respective internal hardenedfield-programmable gate array memory element associated with eachprocessing element, wherein the processing elements are arranged in agrid having a first dimension and a second dimension, wherein a lengthof the grid along the second dimension is greater than a length of thegrid along the first dimension, wherein the processing elements areconfigured to receive the input data along the length of the seconddimension, wherein the input data is written as a ring buffer.
 6. Thedata processing system recited in claim 5, wherein the data processingsystem is configured to process input data arranged in an array, whereineach processing element is configured to receive a respective sub-arrayof the input data, each sub-array having a first sub-dimension lengthand a second sub-dimension length, wherein the sub-dimension lengths areselected such that a height of the array is one processing element. 7.The data processing system recited in claim 1, wherein the processingelements are arranged in a grid having a first dimension and a seconddimension, and wherein a length of the grid along the second dimensionis greater than a length of the grid along the first dimension, whereinthe data processing system is configured to process input data arrangedin an array, wherein each processing element is configured to receive arespective sub-array of the input data, and wherein the processingelements are arranged so as to substantially eliminate redundancy inprocessing data along the second dimension.
 8. The data processingsystem recited in claim 1, the data processing system furthercomprising: an input/output subsystem configured to write input data tothe respective internal hardened field-programmable gate array memoryelement associated with each processing elements, wherein the processingelements are arranged in a grid having a first dimension and a seconddimension, wherein a length of the grid along the second dimension isgreater than a length of the grid along the first dimension, wherein theprocessing elements are configured to receive the input data along thelength of the second dimension, wherein the input data is written as aring buffer, and wherein the processing elements are arranged so as tosubstantially eliminate redundancy in processing data along the firstdimension.
 9. The data processing system recited in claim 1, whereineach of the processing elements is configured to execute one or moreinstructions received from a designated source, a first one of theinstructions including a memory address that references a value storedin an immediate data lookup table located external to the processingelement.
 10. The data processing system recited in claim 1, theprocessing elements being configured to process input data to produce aresult and store the result in an output bus, the result beingnormalized by extracting a contiguous subset of the bits from the outputbus.
 11. A data processing system comprising: a plurality of processingelements, each processing element including a respective internalapplication-specific integrated circuit digital signal processor, eachprocessing element further including an application-specific integratedcircuit memory element; and a plurality of connections between theprocessing elements, wherein each digital signal processor can directlyaccess the application-specific integrated circuit memory element of atleast one other digital signal processor.
 12. The data processing systemrecited in claim 11, wherein each of the processing elements isconfigured to execute one or more instructions received from adesignated source, the execution of the one or more instructions basedon an internal state associated with the processing element.
 13. Thedata processing system recited in claim 11, wherein the plurality ofprocessing elements is arranged in a one-dimensional or two-dimensionalgrid such that each processing element is contiguous with one or moreother processing elements, wherein each processing element can directlyaccess the application-specific integrated circuit memory element ofassociated contiguous processing elements in a single clock cycle. 14.The data processing system recited in claim 11, wherein the dataprocessing system is configured to process input data arranged in anarray, and wherein each processing element is configured to receive arespective sub-array of the input data.
 15. The data processing systemrecited in claim 11, the data processing system further comprising: aninput/output subsystem configured to write input data to the respectiveapplication-specific integrated circuit memory element associated witheach processing element, wherein the processing elements are arranged ina grid having a first dimension and a second dimension, wherein a lengthof the grid along the second dimension is greater than a length of thegrid along the first dimension, wherein the processing elements areconfigured to receive the input data along the length of the seconddimension, wherein the input data is written as a ring buffer.
 16. Thedata processing system recited in claim 15, wherein the data processingsystem is configured to process input data arranged in an array, whereineach processing element is configured to receive a respective sub-arrayof the input data, each sub-array having a first sub-dimension lengthand a second sub-dimension length, wherein the sub-dimension lengths areselected such that a height of the array is one processing element. 17.The data processing system recited in claim 11, wherein the processingelements are arranged in a grid having a first dimension and a seconddimension, and wherein a length of the grid along the second dimensionis greater than a length of the grid along the first dimension, whereinthe data processing system is configured to process input data arrangedin an array, wherein each processing element is configured to receive arespective sub-array of the input data, and wherein the processingelements are arranged so as to substantially eliminate redundancy inprocessing data along the second dimension.
 18. The data processingsystem recited in claim 11, the data processing system furthercomprising: an input/output subsystem configured to write input data tothe respective application-specific integrated circuit memory elementassociated with each processing elements, wherein the processingelements are arranged in a grid having a first dimension and a seconddimension, wherein a length of the grid along the second dimension isgreater than a length of the grid along the first dimension, wherein theprocessing elements are configured to receive the input data along thelength of the second dimension, wherein the input data is written as aring buffer, and wherein the processing elements are arranged so as tosubstantially eliminate redundancy in processing data along the firstdimension.
 19. The data processing system recited in claim 11, whereineach of the processing elements is configured to execute one or moreinstructions received from a designated source, a designated one of theinstructions including a memory address that references a value storedin an immediate data lookup table located external to the processingelement.
 20. The data processing system recited in claim 11, theprocessing elements being configured to process input data to produce aresult and store the result in an output bus, the result beingnormalized by extracting a contiguous subset of the bits from the outputbus.